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SPECIFICATION 



1. Title of the Invention 

Device for recognizing position of vehicle operator 

2. Claims 

1 Device for recognizing position of a vehicle operator, comprising: 
light emitting means for illuminating a vehicle operator; 

image sensing means for receiving reflected light from said operator illuminated 
by said light emitting means, and detecting as image data at least the facial region of said 
operator; 

unique point location sensing means for sensing the location of a unique point in a 
target region of the operator [represented] in said detected image data; and 

location recognizing means for [recognizing] the location of a target region with 
reference to a location having the highest degree of correlation, [said correlation] 
calculated by comparing data for proximity of said sensed unique point location of a 
target region in said image data with data of a standard pattern for said target region 
stored in memory, while varying the position of data of said standard pattern on said 
image data. 

2 Device for recognizing position of a vehicle operator according to claim 1 
wherein said target region is the nose of the operator, and the unique point of the target 
region is the nose tip of the operator. 

3 Device for recognizing position of a vehicle operator according to claim 1 or 2 
wherein the light emitting means emits infrared light, and the image sensing means senses 
infrared light. 

4 Device for recognizing position of a vehicle operator according to any of claims 1 
to 3 wherein the image sensing means is disposed on the front passenger side of said 
vehicle so that the driving posture of the vehicle operator is detected as a side view 
image. 
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5 Device for recognizing position of a vehicle operator according to any of claims 1 
to 4 wherein the image sensing means comprises a solid [-state] two-dimensional imaging 
element. 

3. Detailed Description of the Invention 

(Field of Industrial Utilization) 

The present invention relates to a device for recognizing position of a vehicle 
operator, which captures the driving posture of the vehicle operator in the form of an 
image in order to recognize the position of the vehicle operator within the vehicle cabin, 
and more particularly relates to a vehicle operator position recognizing device designed to 
accurately recognize the location of a target region [of the body] of the vehicle operator. 
(Prior Art) 

Rapid developments in electronic devices in recent years have been accompanied 
by ongoing development of new devices for the purpose of improving vehicle handling, 
comfort, and accident avoidance. One such [kind of device] being contemplated [is one 
capable of] recognizing the driving position of the vehicle operator, for example, the 
position of the driver's eyes or position of the head, and automatically adjusting the 
position and angle of the rearview mirror, headrest, air vents, steering wheel and so on; or 
of sensing if the driver is falling asleep while driving, by detecting periodic change in the 
position of the driver's eyes or position of the head, and issuing an alert to the driver. In 
order to perform such control, a driving position recognizing device is needed in order to 
recognize the driving posture of the vehicle operator; [such a device] could conceivably 
comprise an image sensing portion, consisting of a camera or the like, for imaging the 
driving posture of the vehicle operator and screening the image, and an image processing 
portion, comprising a microcomputer or the like, for analyzing the screened image by 
means of so-called "image processing" to detect on the image a predetermined target 
position, such as the position of the driver's eyes or head, and calculate the position of 
that portion within the vehicle cabin, enabling [the device] to recognize the position of 
the driver. 
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(Problem the Invention Attempts to Solve) 

In order to automatically adjust the position and angle of the rearview mirror, 
headrest, air vents, steering wheel and so on, or to detect periodic change in the position 
of the driver's eyes or position of the head, and issue an alert to the driver in the manner 
mentioned previously, it will be necessary to precisely determine the location of a specific 
region of the operator, such as the location of the eyes, or the location of the nose. 

It is therefore an object of the present invention to provide a device for 
recognizing the position of a vehicle operator, capable of recognizing with precision a 
target region of the vehicle operator. 
(Means for Solving the Problem) 

The present invention, illustrated in Fig. 1, provides a device for recognizing 
position of a vehicle operator, comprising: 

light emitting means II for illuminating a vehicle operator I; 

image sensing means III for receiving reflected light from said operator I 
illuminated by said light emitting means II, and detecting as image data at least the facial 
region of said operator; 

unique point location sensing means IV for sensing the location of a unique point 
in a target region of the operator I [represented] in said detected image data; and 

location recognizing means V for [recognizing] the location of a target region 
with reference to a location having the highest degree of correlation, [said correlation] 
calculated by comparing data for proximity of said sensed unique point location of a 
target region in said image data with data of a standard pattern for said target region 
stored in memory, while varying the position of data of said standard pattern on said 
image data. 

(Description of the Embodiments) 

A first embodiment of the invention is described hereinbelow with reference to 
the accompanying drawings. 

Fig. 2 is a perspective view showing a vehicle instrument panel and surrounding 
area, having installed therein a driving position recognizing device according to this 
embodiment. Symbol 1 denotes the driver, 2 denotes the driver's seat, and 3 denotes the 
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instrument panel. [At a location] on instrument panel 3 diagonally leftward in front of the 
driver's seat 2, i.e., forward of the front passenger seat, are disposed a light emitting 
portion 5 that directs light onto the upper torso of the driver 1 from the left, and an image 
sensing portion 6 that captures the upper torso of the driver 1 as a two-dimensional 
image. 

Here, light emitting portion 5 corresponds to the aforementioned light emitting 
means II, and consists of an infrared strobe that emits infrared light so that the driver 1 is 
not blinded by the emitted light; as shown in Fig. 3, the device comprises an infrared 
emitter 5a, a lens 5b for directing the infrared beam from infrared emitter 5a onto a wide 
area of the driver, an infrared filter 5c that passes infrared light while blocking visible 
light, and a case 5d for housing the infrared emitter 5a, lens 5b, and infrared filter 5c, 
installed at a predetermined location in the instrument panel 3. 

Image sensing portion 6 corresponds to the aforementioned image sensing means 
HI; as shown in Fig. 4, the device is a solid [-state] camera comprising an infrared filter 
6a that passes infrared light, a lens 6b having a focal distance of 12.5 mm for example, 
for focusing an image of the upper torso of the driver 1 onto the image plane of a solid 
[-state] imaging element 6e, described later, a liquid crystal shutter 6c for controlling the 
amount of light [entering], a phototransistor 6d for sensing the amount of light passing 
through liquid crystal shutter 6c, used to automatically adjust the aperture of the liquid 
crystal shutter 6c, a MOS solid [-state] imaging element 6e comprising a photodiode array 
and switching circuitry, for converting an image focused onto its image plane through the 
infrared filter 6a, lens 6b, and liquid crystal shutter 6c into an electrical signal by means 
of a switching scan, and case 6f for housing the aforementioned elements, and attached to 
the instrument panel 3. 

The following description of the overall arrangement of the driving position 
recognizing device of the present embodiment is made with reference to the block 
diagram shown in Fig. 5. 

As shown in the drawing, the recognition device comprises, in addition to the 
aforementioned light emitting portion 5 and image sensing portion 6, an image processing 
portion 8 for processing the image of driver 1 captured by image sensing portion 6, and 
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recognizing a location on [the body of] the driver (in this embodiment, the location of the 
nose). This image processing portion 8 comprises an image input portion 10 capable of 
controlling image sensing portion 6 as well as converting the image signal from image 
sensing portion 6 into a digital signal, and temporarily storing the converted digital signal 
as image data; a strobe signal output portion 12 for outputting a strobe signal that causes 
the light emitting portion 5 to emit light; a central processing unit (CPU) 14 for executing 
a series of location recognition processes, namely, detecting the location of a unique point 
in a target region of the facial region of the driver (in this embodiment, the location of the 
tip of the nose) from the A/D converted image data stored in image input portion 10, and 
recognizing the location of the nose of driver 1 from the location of this unique point; 
read-only memory (ROM) 16 having prestored therein data and a control program for 
execution of the location recognition processes by CPU 14; random access memory 
(RAM) 18 for temporary read/write operations on data used for executing arithmetic 
processes; a bus line 20 interconnecting the above components to enable transfer of image 
signals and control signals therebetween; and a power circuit 22 for supplying power to 
the above components. The image input portion 10 comprises a sync signal generating 
circuit 10a for generating a vertical sync signal and horizontal sync signal for image 
sensing portion 6; an A/D conversion circuit 10b for converting the image signal from 
image sensing portion 6 into image data consisting of a digital signal representing 
brightness of for example, 256 horizontal x 240 vertical pixels per frame; and an image 
data memory circuit 10c capable of storing the preceding two frames of converted image 
data. 

The location recognition process executed by image processing portion 8 is now 
described in greater detail with reference to the flow chart shown in Fig. 6. This process 
is executed when a request for position recognition of driver 1 is input by means of a 
periodic signal, signal from a switch, or the like; as noted, in the present embodiment, 
[the process] recognizes the location of the nose of driver 1. 

Once the process illustrated in the drawing is initiated, first, Step 101 is executed, 
to perform an image data readout process for outputting a strobe signal to light emitting 
portion 5 so that image sensing portion 6 senses an image [illuminated by] the strobe of 
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light emitting portion 5, and stores the sensed image in image input portion 10, in the 
form of image data G consisting of a digital signal. 

Here, image input portion 10 comprises a sync signal generating circuit 10a, an 
A/D conversion circuit 10b, and an image data memory circuit 10c; since image sensing 
portion 6 can be controlled according to a vertical sync signal and horizontal sync signal 
output by sync signal generating circuit 10a, the image signal output by image sensing 
portion 6 sequentially read out, and the image signal having been converted to a digital 
signal by A/D conversion circuit 10b stored as image data G in image data memory 
circuit 10c, it is a simple matter to execute this image data readout process, simply by 
activating the image input portion 10 so that the light emitting portion 5 emits light in 
sync with the vertical sync signal output by sync signal generating circuit 10a. That is, as 
shown in Fig. 7, the process involves outputting a strobe signal S to light emitting portion 
5 once a predetermined time interval AT has elapsed after time to at which the vertical 
sync signal rises, causing light emitting portion 5 to emit light during the vertical 
blanking interval Atj, and reading out the image produced by the emitted light as image 
data G. 

As noted, the A/D converted image data G stored in image input portion 10 
represents the image from image sensing portion 6 in terms of brightness of 256 
horizontal x 240 vertical pixels, storing the brightness G (x, y) for each pixel as shown in 
Fig. 8. 

Once the light emitting portion 5, image sensing portion 6, and image input 
portion 10 have been activated in Step 101 to read out image data G in image input 
portion 10, the system proceeds to execute Step 102, in which there is performed a bright 
spot search process involving designation of a predetermined target area in image data G, 
namely, the zone indicated by symbol 30 in Fig. 9, as the nose zone, searching for pixels 
having brightness above a certain level, present at the leftmost side of this zone, and 
placing the locations of these in memory. This process is utilized when detecting the 
position of the nose of driver 1 in the subsequent step 103, to limit the search range for 
the nose; the pixel position calculated [in this process] serves as the base point for 
searching for the nose. This bright spot search process can be carried out using as the 
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starting point a pixel situated to the left side of the nose zone 30 shown in Fig. 9, with the 
search proceeding in the vertical direction, placing in memory the locations of pixels 
having brightness above an initial predetermined level. 

Once this bright spot search process has been executed, and locations of pixels 
having brightness above the predetermined level, present to the leftmost side of the nose 
zone, have been calculated, the system, in the next step 103, now searches for the nose 
using this position as the base point, and performs a process to determine the location of 
the nose. This process involves examining the degree of correlation of image data in 
proximity to the pixels calculated in the bright spot search, with a predetermined standard 
pattern for a nose like that shown in Fig. 10, and designating the location of the pixel 
having the highest correlation value as the location where the nose is present, to 
determine the location of the nose. 

Specifically, while [designating] each of a total of 121 pixels [in an area] centered 
on the location G(x, y) of a pixel calculated in Step 102 and [extending] 5 pixels each 
way in the vertical and horizontal directions, as shown in Fig. 1 1, as the center location m 
(0, 0) in the nose standard pattern shown in Fig. 10, correlation values between image 
data and the standard pattern are computed, and the pixel location having the highest 
value is recognized as the nose location G (hx, hy). The center location m (0, 0) in the 
nose standard pattern shown in Fig. 10 is predetermined as the portion corresponding to 
the nose; correlation values may be calculated using the equation: 

£G(X + i, Y+j)xm(i,j) 
Regarding values for i and j, since the nose standard pattern consists of data [for an area 
extending] 16 [pixels] above, 7 below, 1 to the left, and 6 to the right with respect center 
m (0, 0), [a range of] integers from (-1) to (+6) for i, and of (-16) to (+7) for j, are 
acceptable. 

Since correlation values with respect to center location m (0, 0) of the nose 
standard pattern are calculated for a total of 121 pixels [in an area] centered on the 
location G(x, y) of a pixel calculated in Step 102 and [extending] 5 pixels each way in the 
vertical and horizontal directions as shown in Fig. 11, the values of X and Y in the 
preceding equation will be integers [in the range] ±5 for X and ±5 for Y; correlation 
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values for the maximum of 121 combinations of X and Y are calculated, and the pixel G 
(X, Y) having the greatest value in the image data is designated as the nose location G 
(hx, hy). 

As will be apparent from the preceding description, in the vehicle operator 
position recognition device of this embodiment, a standard pattern for the target region is 
employed during recognition of the target region, whereby the driving position may be 
recognized more accurately, without erroneous recognition of the location of the target 
region. 

A second embodiment of the invention is now described with reference to the 
accompanying drawings. 

In this [second] embodiment, based on the location of the nose of the driver 1 
recognized [in the manner described in] Embodiment 1, the location of a unique point in 
a target region excluding the nose of the driver 1 (in this example, the center of an eye) 
and the location of a target region (in this example, the eye) are recognized. 

The location recognition process executed in this embodiment is now described in 
greater detail with reference to the flow chart shown in Fig. 12. Step 201, Step 202, and 
Step 203 in this embodiment are analogous to Step 101, Step 102, and Step 103 in 
Embodiment 1, and therefore will not be described again. 

Once the nose location G (hx, hy) has been ascertained by sequentially executing 
Step 201, Step 202, and Step 203, the system proceeds to Step 104, in which the location 
of the eye is now calculated on the basis of the nose location G (hx, hy). Here, when 
calculating the location of the eye, a pixel location G (hx+12, hy -13) 12 pixels to the 
right of and 13 pixels above the nose location G (hx, hy) is calculated, and the degree of 
correlation of the image data to a standard pattern for the eye predetermined for the 
vicinity of this location, shown in Fig. 13, is examined to determine the eye location 
G(mx, my). The determination process is analogous to the nose position determination 
process in Step 103 of Embodiment 1, and will not be described again here. 

Next, in Step 205, on the basis of the eye location G (mx, my) in image data G 
calculated in the preceding Step 204, there is executed a process for computing the three- 
dimensional location of the eye of driver 1 within the vehicle cabin, on the assumption 
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that there is no left or right movement of the driver L Specifically, this process which, on 
the assumption that there is no movement of the driver 1 in the left/right (Z) direction 
shown in Fig. 14, posits an experimentally-derived coordinate system based on a 
reference location P for the eye (left eye) of the driver in the vehicle and calculates a 
three-dimensional location M of the eye (left eye) of the driver in terms of the coordinates 
(X, Y, O), may be carried out readily by means of pre-storing in memory the location G 
(px, py) of a reference point P in the image data, and calculating deviation of the eye 
position G (mx, my) from this reference location G (px, py), since the image sensing 
portion 5 is fixed. 

As will be apparent from the preceding description, in the vehicle operator 
position recognition device of this embodiment, a standard pattern for the target region is 
employed during recognition of the target region, whereby the driving position may be 
recognized more accurately, without erroneous recognition of the location of the target 
region. 

In this [second] embodiment, the location of the driver's eye is not calculated 
directly from image data,. but rather is detected by first detecting the nose, with the nose 
designated as the target region and the location of the tip of the nose projecting out from 
the face -which is the easiest region to detect in the image data- as the unique location in 
the target region; and then with the eye designated as the target region and the center of 
the eye as the unique location in the target region, detecting the location of the eye based 
on the previously calculated location of the nose, whereby the driving position (i.e. 
location of the left eye) can be recognized readily. 

In the present embodiment, the location of the eye (more precisely, the location of 
the left eye) of the driver 1 is recognized by way of the driving position of the vehicle 
operator; however, a three dimensional location could instead be recognized using, for 
example, the location of the nose of the driver 1 as the driving position, so that the most 
appropriate driving position may be recognized with reference to processes occurring 
after the position of the vehicle operator has been recognized, namely, the process of 
automatically adjusting the rearview mirror, automatically adjusting steering wheel angle, 
and the like. Recognition of the location of the eye of the driver 1 may be utilized in 
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automatically adjusting the rearview mirror, sensing if the driver is falling asleep, and so 
on. 

In Embodiments 1 and 2, light emitting portion 5 corresponds to light emitting 
means II, image sensing portion 6 to image sensing means EL, and image processing 
portion 8 to unique point location sensing means IV and location recognizing means V, 
with image sensing means IE consisting of an image sensing portion 6 that in the 
preceding examples comprises a MOS solid-state imaging element; however, a CCD or 
other imaging element employed in so-called solid-state camera could be used instead, or 
a camera tube could be used. 

In Embodiments 1 and 2, a single image sensing portion 6 is provided, and the 
driving position of the driver is recognized using image data of the driver taken from one 
direction, on the assumption that there is no left or right movement of the driver; 
however, image sensing portions could instead be disposed at two different locations, 
using the principle of triangulation to recognize the driving position of the driver, which 
would enable left or right movement of the driver to be detected as well, affording greater 
accuracy in recognition of the driver's 3-dimensional position. 

In Embodiments 1 and 2, an infrared strobe is employed as light emitting means 
n, but continuous light such as an incandescent bulb could be used instead. 

In Embodiments 1 and 2, the bright spot search process has as its starting point the 
upper left edge [of the image], with the search proceeding in the vertical direction; 
however, a search priority ranking could be assigned to all pixels in the nose zone, and 
searched according to this ranking. 

In Embodiments 1 and 2, the nose zone is rectangular, but could instead be round 
or triangular. In this case, effective searches can be carried out by concomitant use of the 
aforementioned method of assigning search priority rankings to pixels. 

Image data obtained from image sensing means IE, rather than being processed 
directly for position recognition, could instead be subjected to an edge sensing process or 
horizontal differentiation process, and position recognition performed on the resultant 
image data, [which has the advantage that] the more distinct features of the image data 
afford greater accuracy in position recognition. 
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(Effects of the Invention) 

As set forth in detail hereinabove, the vehicle operator position recognition device 
of the present invention is designed to recognize the position of the vehicle operator by 
comparing data for the proximity of a unique point, situated in a target region of image 
data obtained from image sensing means, with a standard pattern for the target region 
stored in memory. Accordingly, the target region of the vehicle operator can be 
recognized with a high degree of accuracy, [and the driver's position as determined 
thereby] is useful for automatically adjusting the position and angle of the rearview 
mirror, headrest, air vents, steering wheel and so on, or for sensing if the driver is falling 
asleep while driving, by detecting periodic change in the position of the driver's eyes or 
position of the face, and issuing an alert to the driver. 
4. Brief Description of the Drawings 

Fig. 1 is a block diagram showing the arrangement of the invention; Figs. 2 to 14 
illustrate the embodiments of the invention, wherein Fig. 2 is a perspective view of an 
instrument panel incorporating an embodiment of the invention; Fig. 3 is an illustration of 
light emitting portion 5; Fig. 4 is an illustration of image sensing portion 6; Fig. 5 is an 
overall schematic of the driving position recognition device; Fig. 6 is a flow chart 
showing operation of image processing portion 8 of Embodiment 1; Fig. 7 is a timing 
chart of the image data G readout process; Fig. 8 is an illustration of illuminated image 
data G stored in image input portion 10; Fig. 9 shows an image represented by image data 
G; Fig. 10 shows data representing a standard pattern for a nose; Fig. 11 illustrates a nose 
location sensing process; Fig. 12 is a flow chart showing operation of image processing 
portion 8 of Embodiment 2; Fig. 13 shows data representing a standard pattern for an eye; 
and Fig. 14 describes the three-dimensional location of an eye, wherein (a) is a plan view 
of driver 1 and (b) is a side view of driver 1. 
II ... light emitting means 
m ... image sensing means 

IV ... unique point location sensing means 

V ... location recognizing means 
5 ... light emitting portion 
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6 ... image sensing portion 
8 ... image processing portion 
10 ... image input portion 
14 ... CPU 

Agent: ADACHI Tsutomu, Patent Attorney 

Fig. 1 
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Fig. 4 
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Fig. 5 



/ U 



U 













$ 15 <S 3 33*7 61* ! 






6 











2^ 



•70a 

m 

v— 



Z^> ROM — ;s 



""' t— <fc rai 



« iff. El ?fr 



Key: 5 light emitting portion, 6 image sensing portion, 8 image processing portion, 10 
image input portion, 10a sync signal generator circuit, 10b A/D conversion circuit, 10c 
image data memory circuit, 22 power circuit 



15 



Fig. 6 
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Fig. 9 
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Fig. 11 
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Fig. 12 
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